Developers and quality assurance testers often rely on manual testing to test accessibility features throughout the product lifecycle. Unfortunately, manual testing can be tedious, often has an overwhelming scope, and can be difficult to schedule amongst other development milestones. Recently, Large Language Models (LLMs) have been used for a variety of tasks including automation of UIs, however to our knowledge no one has yet explored their use in controlling assistive technologies for the purposes of supporting accessibility testing. In this paper, we explore the requirements of a natural language based accessibility testing workflow, starting with a formative study. From this we build a system that takes as input a manual accessibility test (e.g., ``Search for a show in VoiceOver'') and uses an LLM combined with pixel-based UI Understanding models to execute the test and produce a chaptered, navigable video. In each video, to help QA testers we apply heuristics to detect and flag accessibility issues (e.g., Text size not increasing with Large Text enabled, VoiceOver navigation loops). We evaluate this system through a 10 participant user study with accessibility QA professionals who indicated that the tool would be very useful in their current work and performed tests similarly to how they would manually test the features. The study also reveals insights for future work on using LLMs for accessibility testing.

Deque Systems, a leading software company specializing in digital accessibility, continues to redefine automated accessibility testing by leveraging Machine Learning technology in its axe Pro beta. In an industry first, Deque has successfully integrated Machine Learning technology to perform powerful visual analyses within axe Pro's automated and intelligent guided testing, which significantly reduces the amount of manual work required to identify and fix accessibility issues. Catching these issues quickly and easily is a crucial step to ensure that websites and apps are accessible to all people, including those with disabilities. "Much of accessibility testing involves determining whether digital content is accurately conveyed to assistive technologies and the users who rely on them to access that content," comments Preety Kumar, CEO, Deque Systems. "By leveraging Machine Learning technology, we've continued to automate many legacy manual testing efforts, drastically reducing testing costs and making better use of a developer's time."